fix(ai): fix unsubstituted template vars + phantom tool in prompts (D1)#1259
Merged
Conversation
…xample (D1) The queries.txt constraint (included by every mode) references $query_types and $output_types_reference, but get_system_prompt only substituted output_types_reference in chat mode and query_types nowhere — so the rendered system prompt leaked literal $query_types / $output_types_reference to the LLM. Substitute both for all modes, derived from FINDING_TYPES (build_query_types / build_output_types_reference) so they can't drift from the registry. Also fix the phantom run_query tool in common.txt's <correct> example: the real tool is query_workspace (TOOL_ACTION_MAP query_workspace -> query), and the example JSON was malformed. Corrected to a valid query_workspace call. Adds regression tests asserting rendered prompts for every mode contain no unsubstituted $query_types/$output_types_reference and reference query_workspace rather than run_query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finding D1 — Prompt template/tool drift (P4 Pertinence).
Problem
constraints/queries.txt(included by every mode) references$query_typesand$output_types_reference, butget_system_promptonly substitutedoutput_types_referencein chat mode andquery_typesin no mode. Result: the rendered system prompt leaked literal$query_types(all 3 modes) and$output_types_reference(attack + exploit) to the LLM.constraints/common.txt<correct>example taughtrun_query(...)— a tool that does not exist. The real tool isquery_workspace(TOOL_ACTION_MAP["query_workspace"] == "query"). The example JSON was also malformed (unbalanced braces, wrong_typeshape).Fix
secator/ai/prompts.pyget_system_prompt: build one substitution dict withquery_types=build_query_types()andoutput_types_reference=build_output_types_reference()for all modes (library_reference/path_vars still only for attack/exploit). Values derive fromFINDING_TYPES, so no hardcoded list to drift. Uses existingsafe_substitute.secator/ai/prompts/constraints/common.txt:15:run_query({...})->query_workspace(query={"_type": "vulnerability", "severity": {"$in": ["high", "critical"]}}), matching thequery_workspaceexamples already inqueries.txt.Value sources
$query_types<-build_query_types()(comma-joinedcls.get_name()overFINDING_TYPES).$output_types_reference<-build_output_types_reference()(same registry).secator/ai/tools.pyTOOL_ACTION_MAP(read-only).Tests
tests/unit/test_ai_prompts.py: 38 -> 41 passed. Added 3 regression tests asserting every mode renders with no$query_types/$output_types_reference, thatquery_typesrenders to real registry names, and that prompts referencequery_workspacenotrun_query.Rendered-prompt check before/after:
$query_types$output_types_referencerun_query🤖 Generated with Claude Code